MAGAZINE 





FOR NOVICE AND deltas 2 oe 


on i ies ae oe . 


; / CARLOS NEIRA _ 
jo. 


Cae 


ema 
AND TROUBLESHOOTING 





Start saving your time today and find out how usefull skills you can learn with Carlos. 


In this workshop you will see real life situations, where debugging skills will save you time, headaches 
and possibly find a solution with minimal amount of effort. 


Debugging/Troubleshooting is a really useful skill when you are working in maintaining legacy applications, doing some small 
incremental changes to an old code base, where the code has been touched by so many hands over the years and it is 
becoming really a mess. So, management has decided that the code works as it is and you are not allowed to change it all over 
“the right way ™”, 
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About the Instructor 


Carlos Antonio Neira Bustos has worked several years as 
a C/C++ developer and kernel porting and debugging 
enterprise legacy applications. He is currently employed 
as aC developer under Z/OS, debugging and 
troubleshooting legacy applications for a global financial 
company. Also he is engaged in independent research on 
affective computing. 





Course format: 


The course is self-paced — you can visit the training whenever you want and your content will be there. 
Once you’re in, you keep access forever, even when you finish the course. 

There are no deadlines, except for the ones you set for yourself. 

We designed the course so that a student will need about 18 hours of work to complete the training. 


Your time will be filled with reading, videos, and exercises. 


— ale. https://osdmag.org/course/application-debugging-and-troubleshooting-2/ 
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Introduction to the GDB 


The Basics 


To be able to inspect more easily a program, we need to have the symbol table available for the program 
we intend to debug. This is accomplished using the —g flag of the compiler we are going to use (we 
could also debug it without the —g flag, but it is really cumbersome sometimes). In our case, we will use 
FreeBSD 10 as the platform and the clang compiler that comes with it. 


After the program has been compiled using the —g flag, we are able to peek inside it using the gdb 
debugger to start a debugging session. All you need to do is type: 


# gdb <program name> 


And we will see a (gdb) prompt. That means that we are ready to start typing gdb commands. 


aoe alte ee SU ee ei Se S| 

INU gdb 6.1.1 [FreeBSD] 

“opyright 2004 Free Software Foundation, Inc. 

DB is free software, covered by the GNU General Public License, and you are 
welcome to change it and/or distribute copies of it under certain conditions. 


ype “show copying” to see the conditions. 

here 1S absolutely no warranty for GDB. Type “show warranty” for detatls. 
his GDB was configured as “amd64-marcel-Treebsd”... 

‘gdb) 





Or if the program we need to debug is currently running, we must type: 
#gdb 
#(gdb) attach <pid of running program> 


Let’s start with some basic commands and inspect a running application. For this example, | have 
selected this application http://freeciv.wikia.com/wiki/Main_Page. 


“Freeciv is a Free and Open-Source empire-building strategy game inspired by the history of human 
civilization. The game commences in prehistory and your mission is to lead your tribe from the Stone 
Age to the Space Age...” 


We will inspect the game structures at runtime with gdb. So, let’s follow these steps: 


Edit /etc/make.conf and add the line WITH_DEBUG=yes (this will not strip your binaries. You will have 
the symbol table and also add the debug flags to the compiler when compiling the sources of your 
ports). 


Install freeciv from ports. 


Start the freeciv server and client (freeciv-server and freeciv-gtk2). 


Join your local game. 


freeciv (om trueos) J & =) x] 





Game Government View Orders Reports Editor Help 








Figure 1. Joined the local game 
Now, we will use our first gdb command: 


# gdb /usr/local/bin/freeciv-server 


reeciv-server 


right 2004 Free Software Foundatlion, Inc 


is free software, 
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warranty” for details. 


‘amd64-mgarcel-freebsd".. 


Te leer met)wtle lee local /bin/freeciv-server 

New LWP 100405] 

New Thread 805406400 (LWP 100405/free 

his 1s the server for Freeciv version 2. 

‘ou can learn a lot about Freeciv at http://www. freeciv.org 

eae jer program has player authentication support, but it's 
; tS. 
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As we don’t know anything about how Freeciv works, we will press CTRL-C, this will interrupt the 
program and we will take from there. For starters, let’s interrupt and see where we are. If we want to 
continue the execution, we type ‘continue’ or ‘c’ 

rogram received signal 


Switching to Thread 
x0000000801dd606a in 


Sane 
805406400 


Jaane 


interrupt. 
(LWP 100405 
cae eee 


Treeciy 
Dili leet ee 
in select ¢) from 
2 in select () from 
d203 in fc_select (n readfds-Ox7fffttfffdebs, 
thie) TAO POOR O Eiht). Pee) oe het) ee Oe eet b yt. pean) |) eee ee Pls 
erat a O8fa58a in server_sniff_all_input © at sernet.c:686 
0x00000C€ ts in Srv_running (©) at srv_main.c:2317 
in Sfv_main () at Srv_main.c:277 
in main (Carac aroveOx7fffffffdass) at civserver.c:453 


lib/libc.so.7 
lib/libthr.so. 3 
writefds-Ox7fftfttttfd638, 


1x000000000040276Ca 





Here is a screenshot from the client program freeciv-gtk2. We need to join our local game as we are 
going to debug the server. 





eee es ea 


jew Coacne ermect to Network Garme 
art Scone are annect to Garning Jorn 
Loed Saved Gaerne pL, Quit 


rogram received signal SIGINT, Interrupt. 
W000801dd606a In select () from /1ib/libc.so.7 
ei 
Pelee EEO eRe el cltlce ee eto l-  Oeeas Biya b lt eee, 

UU eRe Feit Pe eo Oe eat hy A thail ee ee 

000000800dcd203 In fc_select (n-7, readfds-Ox7ftttftfd6es, writefds-Ox7tfrttttd6es, 
exceptfds-Ox7Tfttttfdses, timeout-Ox7Tfttttfdsds) at netintt.c:126 

Ox00000008008Tas&8a in Sserver_sniff_all_input () at sernet.c:686 

O0000080090T 407 In srv_running (©) at srv_main.c:2317 

00000080090e0a4 In Srv_main () at Srv_main.c:2777 

0000000004026ca In main Caracel, argqv-Ox7TTftfttfdass) at civserver.c:453 

] 

7 Dx000000080090T 407 In Ssrv_running (€) at srv_main.c:2317 

ees while (server_sniff_all_input() -- S_E OTHERWISE) { 


ee ee ee 
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are = sniff, re-zero the tiger: (read-out above on next loop) 


(gdb) § 


The #<num> you see are the stackframes, or simply called frames. When your program is started, the 
stack has only one frame, that of the function main. This is called the initial frame or the outermost 
frame. Each time a function is called, a new frame is created. Each time a function returns, frame for 
that function invocation is eliminated. If a function is recursive, there could be many frames for the same 
function. The frame for the function in which execution is actually occurring is called the innermost 
frame. This is the most recently created of all the stack frames that still exist. 


Let’s go into frame 3. To do this, we type either ‘frame 3’ or ‘f 3’. 


i 
conn_list_iterate(game.all_connections, alae si0 met 
if (srvarg.auth_enabled 
&& !pconn->server.is_ closing 
&& pconn->server.status != AS ESTABLISHED) { 
auth_process_ status(pconn); 
; 
} conn_list_iterate_end 


if (S_S RUNNING «= server _state() && game.info.timeout == 
(void) send_server_info_to_metaserver(META_REFRESH); 
return S_E_END_OF_TURN_TIMEOUT; 


con_prompt_o 


if (fc_select(max_desc + 1, &readfs, &writefs, &exceptfs, &tv) == @) 


(void) send_server_info_to_metaserver(META_REFRESH); 
if (game.info.timeout > 
&& S_S RUNNING == server_state() 
&& game.server.phase_timer 
MGs MO ee ae Ot eee 
> game.info.seconds to phasedone)) { 
con_prompt_off(); 
return S E END OF_TURN_TIMEQUT; 


. 





It seems that the server will send us end of turn. Let’s make sure a break point is set as follows: 
the format is > 


<break|b> <source.c>:<line number> (gdb) b sernet.c:695 


Telemed] GM PRM M TH ie cl 
check_for_full_turn_done(); /* HACK: don't wait during AI phases * 
while (server_sniff_all_input() == S_E_OTHERWISE) { 

nothing * 


‘ After sniff, re-zero the timer: (read-out above on next loop) 


3 
3 
(ols) ae 
0x00000008008fa58a in server_sniff_all_input © at sernet.c:686 

if (fc_select(max_desc + 1, &readfs, awritefs, &exceptfs, &tv) == 0) { 


+ ea 
, 


log_debug("sni ffingpackets”) ; 
aia GRC me Re amt Oban |G Oar RR RBG ante OM ir hte 
while (server_smiff_all_input() -~ S_E_OTHERWISE) { 

* nothing * 


ome RC MPEP Od Me eet) 0) 


)8008faS8a in server_smiff_all_input () at sernet.c:686 
oe Gielen ee ree Cee. 2 eae ee 10 en) mt 
(gdb) b sernet.c:695 
ter ae ee 
(gdb) 


(gdb) fF 3 
Ox00000008008Ta58a in server_sniff_all_input ( at sernet.c:686 
ete if (fc_select(max_desc + 1, &readfs, a&writefs, &exceptfs, &tv) == 0) 
Gol me mete 
Md ee 
onnections = false 
- 256 
a: 
ax_desc = 6 
> (ate ee at 
eadfs = {_Tds_bits = 189, 0 <repeats 15 times>}} 
writefs «= {_fds_bits - {0 <repeats 16 times>}} 
xceptfs - {__fds_bits - {88, 0 <repeats 15 times>}} 
v = {tv_sec = 1, tv_usec = 0} 





Typing i lo means info locals which will display all local variables in this frame and their values, which is 
pretty handy. 


Let’s take a look at something easier to see. Sometimes in freeciv, another civilization will try to enter 
into negotiating terms with us. Therefore, looking at the source code, we find the add_clause function in 
the diptreaty.c source code. That function will add a term which will make the other part accept or reject 
our terms. 


/usr/home/cneira/workshop/freeciv-2.4.3/common 





+~eGiptreaty.¢ + 


Add clause to treaty. 


boo! add_clause(struct Treaty “ptreaty, struct player “ptfrom, 
enum clause_type type, int val) 


struct player “pto (ptrom ee oe ei iat 
> ptreaty->pirl : ptreaty->plro); 
struct Clause *pclause;: 
enum diplistate_type ds 
player_dipistate_get(ptreaty->plro, ptreaty->plrl) 


1F (type < O type >= CLAUSE LAST) 
log_errorC"Illegal clause type encountered 
return FALSE; 


CLAUSE ADVANCE && '!valid_advance_by_number(val)) { 


i tthe ee tO <repeats 16 times> 
fds_bits = {88, 0 <repeats 15 
u Ll, tTyv_use 0} 
(aqdb)> b add_clause 
sreakpoint 1 at Ox800cfT9436: file diptreaty.c, 
Gilels Dae: 
ih marl eae 


sreakpoint 1, add_clause (ptreaty-O0x8064b9ee0, pfrom-0x80630dd400, type-CLAUSE_ CEASEFIRE, val«0) 
at diptreaty.c:138 


(gdb) § 





After playing a few minutes, we hit a break point. At this point, we don’t even know which civilization has 
approached us to negotiate terms. 


Now, we could know ahead since we set the breakpoint where the negotiation starts. 


Add clause to 


bool add_clause(struct Treaty “ptreaty, struct player “pfroa, 
enum clause_type type, int val) 


struct Clause “pclause; 
enum dipistate_type ds 
player_dipIistate_get(ptreaty->plir0, ptreaty->pirl1)->type; 


oO 6/1) eat type >= CLAUSE_LAST) { 
log _error("Illegal clause type encountered.”): 





‘eebsd-th Thread 8054064 In: add_clause Line: 138 PC: Ox800cf9436 
vec “g@: +, Skill_tevel AT LEVEL_EASY, rr a | tte iti t 10, science_cost 100, warmth 
Out) a es ae ae ae ewe eae T {1 <repeats 128 times>}}, ai tee LP Atl Se Lele 
was_created « false, 1S_connected « true, current_conn « Ox0, connections «= 0x80682c120, 
Eh ir at {vec "\O" <repeats 15 times>}, wonders (Rese ere ee 
attribute_block = {data - 0x0, length - attribute_block_buffer [data = 0x0, length - O}, 
ti le_known ane 3888, vec 0x805643400 “" tp ete i) » se (status {vec ‘\OO1"}, 












got_first_city - false, private_map - Ox80é 00, really_gi 
iu ‘\O" <repeats 15 times>}, debug 4 wy, adv 0x80543 ar 1 tit. Fa Oe 


’ ‘ } 


0x0}, delegate_to - O* <repeats 47 times>, orig_username )" <repeats 47 times>}, client 
A et {ibits | ee 0x8064d0000 “"}, {bits UP Os eae 
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| assume the negotiation civilization should be in the pfrom pointer. 


Viet ae me ee 
10 = (struct player *) 0x8063dd400 





ee 


To print the variables values, we just type ‘p’. In this case, ‘p’ is a pointer to a player structure. If we 
would like to check the definition of the player structure, we just type ptype pfrom, and the structure 
definition will be displayed. 


ype = c 

struct player_slot *slot; 

char name[48]; 

char username[48]; 

char ranked_username[48]; 

int user_turns; 

_Bool 1s_male; 

struct government “government; 

struct government *target_government; 
struct nation_type *nation; 

--Type <return> to continue, or q <return> to i | 





Now, let’s find out the values of these fields that the civilization demands. Since the pfrom is a pointer, 
we need to use pointer notation to check its contents. 


warmth = 0, Trost = 0, barbarian_typ OT_A (BARIAN, love = i0, 0, 0, 0, 0, 
1 <repeats 123 times>}}, ai = 0x80 (Pee ae eee he el 
Ph et De A ee Rl | 
wonders = {0 <repeats 21 times>, 129, 0 <repeats 178 times>}, attribute_block = {data = 0x0, 
length = 0}, attribute_block_buffer - {data = 0x0, length = 0}, tile_known = {bits = 3888, 
vec = 0x805643800 ""}, rgb = Ox8064alea0, {Server = {status = {vec = "\001"}, got_first_city = true, 
private_map = 0x80652c000, really_gives_vision = {vec = '\O' <repeats 15 times>}, debug = { 
vec = ""}, adv = 0x80543d800, ais = {0x80626T000, Ox0, Ox0}, 
delegate_to = '\0' <repeats 47 times>, orig_username = ' Se | 
tile_vision = {{bits = 257, vec = 0x80652c000 “"}, {bits = 0, vec = 0x0}} 
VC el 1D rom 


Uxs0540/ lle tT 1S - ; <repea 
Nt hed 1) 


username = “Unassigned”, 
iked_username = “Unassigned”, ‘'\O' <repeats 37 times>, user_turns = 10, 1S_male = true, 
nment « 0x805575500, target_government «= Ox0, nation = 0x8068470C8, team = 0x805603fe0, 
is_ready = false, phase_done = false, nturns_idle = 9, 1S_alive = true, revolution_finishes - 
real_embassy = {vec = '\O" <repeats 15 times>}, diplstates = 0x80540c400, city_style = 0, 
cities = 0x80682 Ae PRO eo) cts Poaceae GT |)))) eae a irl) ee ie 


specialists = {0 <repeats 20 times>}, wonders = 0, techs = 0, techout = 2, landarea = 

settiedarea - 4000, population - 20, cities - 2, units = 0, pollution - 0, literacy - 0, bnp - 4, 

mfg = 5, spaceship Pe eho wee 0, units_killed = 0, units_lost 0, game PAP eae 
-Type <return> to continue, or q <return> to quit-- y 





Looking at the player struct , it seems that the leader’s name is Roy Jenkins, and looking at the 
backtrace (bt), the clause of the treaty seems to be “cease fire”. So, we are going to be offered a peace 
treaty. 


To continue executing the program, type ‘next’ or ‘n’ . Something like this will be displayed in the 
diplomacy tab: 
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Game Edt View Select Unit Work Combat Civilization Help 


Jemplars Givankana 
Master Jacques de Molay 
Ternplars 


Gold: ~ dp Add Clause... 
Population: 0 7 Europeans 
Year: 3550 BCE 
Gold: 100 (+0) Fea Chief Roy Jenkins 
Tax: 40 Lux: 0 Ses: 60 


Gold: 0 = dP Add Clause... 
UIE causes: 
GPR oot 


Turn Done 
Legion 


Moves: 1 
Jungle 


c 


Accept treaty J Cancel meeting 


What you cannot see in the screenshot is that | have requested an embassy in return for the cease-fire 
treaty, but here it is: 








freebsd-th Thread 8054064 In: add_clause Line: 143 PC: Ox800cfS 
> 00080098af73 in manage_auto_explorer (punit-0x8054d7900) at autoexplorer.c:396 

do_explore (punit-0x8054d7900) at unittools.c:2447 38 

1f1d5 in srv_running () at srv_main.c:2261 

0a4 In Ssrv_main ©) at srv_main.c:2777 


main Cargcel, argveOx7fffffffdass) at civserver.c:453 


+3 





Glelsp re 
Continuing. 


Game saved as Tfreeciv-T0009-Y-3550-auto. sSav.bz2 


Breakpoint 2, add_clause (ptreaty-O0x8064bD9ee0, prrom-0x8063dd400, 
at diptreaty.c:13 i 
(gdb) 


type-CLAUSE_EMBASSY, val-0) 


Let’s go line by line using next. You could also use the step command but if you use the comman4d, it will 


take you inside a function call instead of just evaluating the function and returning like the next 
command. 
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+--diptreaty .c--------- 


‘an aa te ee! aa he eed ciate 
*“pcla 1 ae 

enum dipistate_type ds 
Se UE Li tema Ree: be Mt eG haa a ee deat eae oe ee hi Be ab 





tm ee) oe egal clause type encountered.”); 
eturn FALSE: 


(type «— CLAUSE_ADVANCE 4& 'valid_advance_b uaber(val)) 
log_error("Illegal tech value %1i in clause.”, val): 
return FALSE; 


clause(type) 
DS_ PEACE && as tls CLAUSE 
te re ce. a’ pe 
ALLIANCE && type «= ( 


CEASEFIRE G& type 


Clete ae 

#0 add_clause (ptreaty-O0x8064b9ee0, pfrom-0x8063dd400, type-CLAUSE_EMBASSY, val at diptreaty.c:141 

Gl eee 

Crisp) 

(adb) list 

Cele) Dee 7 

Ee CLAUSE EMBASSY 

(gdb) ptype CLAUSE EMBASSY 

a7 em ott em ELT: he {CLAUSE ADVANCE, CLAUSE GOLD, CLAUSE_MAP, CLAUSE SEAMAP, CLAUSE 
CLAUSE CEASEFIRE, CLAUSE PEACE, CLAUSE ALLIANCE, CLAUSE VISION, CLAUSE EMBASSY, 

Cele eD, 


nation_rule_name(nation_of_player(ptreaty->piro)), 
nation_rule_name(nation_of_player(ptreaty->pirl))): 
return FALSE: 





we already have embas 
perror("Iilegal embas clause: %S already have embassy with & 
nation_rule_name(nation player(pto)), 
nation_rule_name(nation player(pfrom))): 
return FALSE: 


if (!'gqame.info.trading_gold && type CLAUSE GOLD) { 
return FALSE: 


if ('game.info.trading_tech && type -— CLAUSE ADVANCE) 


return FALSE: 


if (!gqame.info.trading_city && type CLAUSE_CITY) 4 


We are currently at line 143. | just checked what kind of data type was CLAUSE_EMBASSY. It was an 
enum one (Somewhat obvious). 


Using ‘next’ command a couple of times will get us here: 


Keep on typing ‘n’ and we will exit from the function call and arrive to 
handle_diplomacy_create_clause_req 


diplomacy_create_clause_ req { counterparte2Z, QivereZ2, typeeCLAUSE EMBASSY, 
value-0) at dipihand.c:679 
UUM GD tla teeta cam tome ar lee cod al DIPLOMACY _CREATE_CLAUSE_REQ, 
acket-Ox8068d0250, pplayer-Ox8063dbc00, 30) at hand_gen.c: 
( in server_packet_input ( C37580, packet=-0x8068d0250, type=-99) 


In incoming _client_packets (pconn-0x800c37580) at 
in server_sniff_all_input © at sernet.c 
srv_running () at srv_matn. wy 
1 srv_main () at srv_main.c:2777 


A “  & | 
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As we keep on typing ‘next’ , we will arrive at this function call_treaty_evaluate that seems interesting. 
Maybe here the results of rejection or acceptance of conditions are done. As | explained earlier, we can 
step into this one using the step command. 


player_number(pother), giver, type, 
Value); 
disend_packet_diplomacy_create_clause(pother->connections, 
player_number(pplayer), giver, type, 
Value); 
treaty); 
oe SR ae he ae Rh TGsle hate neh tae a 


Cancel meeting. No sanity checking of input parameters, so don't call 
this with input directly from untrusted source. 


static void really_diplomacy_cancel_meeting(struct player “pplayer, 
struct player “pother) 
Sea eee 18 eta e te ee el a Ot he ted 
hme das 15 pee! 
;_ create clause_re } : 0x80087d537 
<return> to quit 
argv-Ox7fffffffda58) at civserver.c:453 
n 
n 
inet 
n 
n 
' eet 
8 ED 
gdb) list 
gdb) § 


* FIXME: Should this be put in a ruleset Somewhere? 
#define TURNS_LEFT 16 


such 18 set for 


Static void call_treaty_evaluate(struct player “pplayer, struct player “*aplayer, 
struct Treaty “ptreaty) 
(pplayer->ai_controlled) { 
CALL_PLR_AT_FUNC(treaty_evaluate, pplayer, pplayer, aplayer, ptreaty); 


| 
j 


RARER ROR COARSE 


Calls treaty_accepted function if such 1s set for AI player. 


Static void call_treaty_accepted(struct player *pplayer, struct player *aplayer, 
struct Treaty “*ptreaty) 


reebsd-th Thread 8054064 In: call_treaty_evaluate : : 0x80087d314 
adb) 

relelep 

relelap 

gdb 

Pll 

elelep 

ela 

rl ae Rete I 

gdb) step 

hy Nel creatine (pplayer=j0x8063dbc00, aplayer=0x8063dd400, ptreaty=0x8064b9ee0) at diplhand.c:75 
gdb) 





Let’s step all the way to get to another point in the program execution. After a couple of steps, we get to 
this point: 


aty being considered. It is all a ques 


ty_evaluate(struct player “pplayer, struct player “aplayer, 
Struct Treaty “ptreaty) 


int total_balance « 0; 

bool only_qifts TRUE: 

enum dipistate_type ds_after - P 
player_dip!istate_get(pplayer, aplayer)->type; 

int given_cities = 0; 


clause_list_iterate(ptreaty->clauses, pclause 
if (s_pact_clause(pclause->t) ) 
ds_after — pact_clause_to_dipistate_type(pc lause->type); 
1 
| 
if (pclause->type —- CLAUSE_CITY && pclause->from -- pplayer) { 
given_cities++; 
; 
; 


} clause_list_iterate_end; 





‘eebsd-th Thread 8054064 In: dai_treaty_evaluate Line: $78 PC: 0x80094a937 


0) at diplhand.c:693 


treaty_evaluate (pplayer-0x8063dd400, aplayer-Ox8063dbc00, ptreaty-0x8064b9ee0) at diplihand.c:75 
bei 


ete a Rees ee Lucey ete le tC tM Pe uch eel atte eae th ete et oct me heme erie ae 
We he: 





So, a quick glance at the source code tells us that the total_balance variable is somewhat important to 
evaluate if a clause is accepted (In our case, we are requesting to give us an embassy). Instead of 
printing this variable multiple times, let's leave it available on the display. 


#(gdb) display total balance 
Then, we set a breakpoint somewhere ahead of advdiplomacy.c:621. We can see that the total_balance 


value is displayed and it is -450, seems bad for our proposal. 
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(given_cities > 0 
7S A ee ee aD given_cities 
PO Ce a oe 
DIPLO_LOG(LOG_DIPL2, pplayer, aplayer, “cannot give last cities"); 
eat 


Accept 1f balance 1s good 
if (total_balance >. 0) { 
handie_diplomacy_accept_treaty_req(pplayer, player_number (ap 
DIPLO_LLOG(LOG_DIPL2, pplayer, aplayer, “balance was good: %d", 
total_balance); 


j 
1 


AIT complains about the treaty which was proposed, unless the AI 
the proposal. 


eG eee gene ee era ee 
eRe ayer, _("*%s (AI)* This deal was not very good for us, %s'"), 
arene hd 
amie kee lek eh cl De 


freebsd-th Thread 8054064 In: dai_treaty_evaluate Line: 621 PC: 0x80094ac13 


Pe ee Oe al) eo 

(gdb) list 

(gdb) b advdipjomacy.c:621 

Breakpoint 3 at 0x80094ac13: file advdiplomacy.c, line 621. 

(gdb) <« 

Continuing. 

Breakpoint 3, dai_treaty_evaluate (pplayer-0x8063dd400, aplayer-Ox8063dbc00, ptreaty-0x8064b9ee0) 
at advdiplomacy.c ) 

1: total_balance 

Ciel. 0 





As we can see, total_balance >=0 is the condition to approve the proposal. The following is a resume 
of the commands used in this session: 
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COMMAND ABBREVIATED FORM | WHAT IT DOES 

info local ilo Prints values and names of all local variables in 
the current 
scope. 

backtrace bt A backtrace is a summary of how your program 


got where it is. It shows one line per frame, for 
many frames, starting with the current executing 
frame (frame zero), 

followed by its caller (frame one), and on up the 
stack. 





frame <frame number>|f <frame number> 


The call stack is divided up into contiguous pieces 
called stack frames, or frames for short; each frame 
is the data associated with one call to one function. 
The frame contains the arguments given to the 
function, the function’s local variables, and the 
address at which the function is executing. 





print <variable> 


p <variable> 


displays the value of the variable 





display <variable> 


disp <variable> 


Will automatically print the value of the variable 
being displayed as long as it is within the scope 














win Win Will enter gdb in tui (text user interface) mode if 
we had not entered in the first place. Default 
layout is source at the top commands at the 
bottom. 
next n Executes next line of code. Will not enter 
functions. You can use as parameter the number 
n <number of next to | Of times to execute next 
perform> 
step Ss Step to next line of code. Will step into a 
s <number of function. 
steps to 
perform> 
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These are really basic commands, but useful. Armed with these commands, you should be able to 
answer a couple of questions using gdb and not just browsing the source code: 


Describe the flow when a city is created, which are the stack frames? 
What is the name of the structure which handles units? 


What is the name of the structure that handles game information? What are the contents of this structure 
at the end of a game? 
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Advanced Inspection of 
Data Structures and 
Variables 


Automating Information Display 


Now that we have used the display command or the print command, it is getting pretty tedious to inspect 
a variable or data structure manually by typing p or display every time we hit a breakpoint we have set. 
There is a command called commands to save us from all this typing. 


First, we set a breakpoint where we want to inspect data automatically. In this case, I’ll check one of the 
city functions. 


(gdb) b city.c:2352 


(gdb) 4 breakpoint keep y 0x0000000800cf1b7b in citizen base mood at 
city.c:2352 


Now, we can type the following: commands <breakpoint number> 
(gdb) commands 4 


Type commands when breakpoint 4 is hit, one per line. Then end with a line 
Saying "end". 


> 


After you have set the instructions to be executed after the breakpoint is hit, you could modify them or 
just erase them like this: 


(gdb) commands 4 


Type commands when breakpoint 4 is hit, one per line. End with a line saying 
W end" 7 


> end 
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Now, if you want to execute something: 

(gdb) commands 4 

Type commands when breakpoint 4 is hit, one per line. End with a line saying "end". 
> printf “Setting city mood for leader: %s”, pplayer->name 

> end 


We can type all the instructions we want to be executed when this breakpoint is hit. Usually, we used 
print to display values, but there is a more powerful function called printf that uses similar format as the 
C-language function 


(gdb) printf “%Ss”, pplayer->name 


As in C’s printf, ordinary characters in the template are printed verbatim, while conversion specification 
introduced by the ‘%’ character causes subsequent expressions to be evaluated, their values converted 
and formatted according to type and style information encoded in the conversion specifications, and then 
printed. 


For example, you can print two values in hex like this: printf "foo, bar-foo = Ox%x, 0x%x\n", foo, bar-foo 


printf supports all the standard C conversion specifications, including the flags and modifiers between 
the ‘%’ character and the conversion letter, with the following exceptions: 


« The argument-ordering modifiers, such as ‘2$’, are not supported. The modifier *’ is not supported for 
specifying precision or width. 


- The “’ flag (for separation of digits into groups according to LC_NUMERIC’) is not supported. The type 


3 aus ‘ 


modifiers ‘hh’, ‘j, ‘t’, and ‘Zz’ are not supported. 


- The conversion letter ‘n’ (as in ‘Y%n’) is not supported. The conversion letters ‘a’ and ‘A’ are not 
supported. 


Note that the ‘Il’ type modifier is supported only if the underlying C implementation used to build GDB 
supports the long long int type. Likewise, the ‘L’ type modifier is supported only if long double type is 
available. 


As in C, printf supports simple backslash-escape sequences such as \n, ‘\t’, ‘\V, ‘\V'", ‘\a’, and ‘\f, that 
consist of backslash followed by a single character. Octal and hexadecimal escape sequences are not 
supported. 


Additionally, printf supports conversion specifications for DFP (Decimal Floating Point) types using the 
following length modifiers together with a floating point specifier. letters: 


‘H’ for printing Decimal32 types. ‘D’ for printing Decimal64 types. 


‘DD’ for printing Decimal128 types. 
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If the underlying C implementation used to build GDB supports the three length modifiers for DFP types, 
other modifiers such as width and precision will also be available for GDB to use. 


In case there is no such C support, no additional modifiers will be available, and the value will be printed 
in the standard way. 


Here’s an example of printing DFP types using the above conversion letters: printf "D32: %Hf - D64: %Df 
- D128: %DDf\n"",1.2345df,1.2E10dd,1.2E1dl 


Dynamically allocated arrays 


Sometimes we will need to take a look at the contents of dynamically allocated arrays (the ones created 
by malloc and calloc system calls). 


For example, we have the usual static memory array: 

char t[8001]; 

It’s easy to display its contents using: 

(gdb) pt 

But what about this one: 

ant. *t7 

t = (int *) malloc ( 8001 * sizeof( int) ); (gdb) pt 

This will give only the address: 

(gdb) p *t 

This will give you the data of the first element in the array, so what is the solution? 
(gdb) p *t@25 

This command will print 25 elements from the array t .The format is pointer@<number of elements. 
Getting information from the symbol table 


When we compiled our program with the —g flag, we instructed the compiler to generate a symbol table 
in Our program binary. The table contains variable names, function names and types. Now, let’s suppose 
we want to know the names of all the functions available, we could use one of the info family commands: 


(gdb) info functions 


This command will print the names and data types of all defined functions. If we want to check only the 
function names matching a regexp ,we use the command: info functions <regexp> 
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For example: 


(gdb) info functions city 


The above command matches all functions that have city string in their name. You must use grep regexp 
not perl’s regexp . The same goes with variables with the command: 


(gdb) info variables 


Prints the names and data types of all variables that are declared outside of functions (not the local 
variables). 


Also, the same syntax for info variables regexp 
(gdb) info variables city 


Prints the names and data types of all variables (except for local variables) whose names contain a 
match for regular expression regexp. 


(gdb) info address symbol 


Describes where the data for symbol is stored. For a register variable, this says which register it is kept 
in. For a non-register local variable, this prints the stack-frame offset at which the variable is always 
stored. Note the contrast with ‘print &symbol', which does not work at all for a register variable. For a 
stack local variable, it prints the exact address of the current instantiation of the variable. 


(gdb) whatis exp 


Prints the data type of expression exp. exp is not actually evaluated, and any side-effecting operations 
(such as assignments or function calls) inside it do not take place. Any kind of constant, variable or 
operator defined by the programming language you are using is valid in an expression in GDB. 


(gdb) whatis 
Prints the data type of $, the last value in the value history. 
(gdb) ptype typename 


Prints a description of data type typename. typename may be the name of a type, or for C code it may 
have the form ‘class class-name’, ‘struct struct-tag’, “union union-tag' or ‘enum enum-tag’. 


(gdb) ptype exp ptype 


Prints a description of the type of expression exp. ptype differs from whatis by printing a detailed 
description, instead of just the name of the type. For example, for this variable declaration: 


struct example {double dtype;, float ftype} exl; 


The two commands give this output: 
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(gdb) whatis exl type = struct example (gdb) ptype exl 
type = struct example { double dtype; float ftype; 


} 


As with whatis, using ptype without an argument refers to the type of $, the last value in the value 
history. 


(gdb) info types regexp 


Prints a brief description of all types whose name matches regexp (or all types in your program, if you 
supply no argument). Each complete typename is matched as though it were a complete line; thus, *i 
type value’ gives information on all types in your program whose name includes the string value. But i 
type “value$' only gives information on types whose complete name is value. This command differs from 
ptype in two ways: first, like whatis, it does not print a detailed description; second, it lists all source files 
where a type is defined. 


(gdb) info source 


Shows the name of the current source file--that is, the source file for the function containing the current 
point of execution--and the language it was written in. (gdb) info sources 


Prints the names of all source files in your program for which there is debugging information, organized 
into two lists: files whose symbols have already been read, and files whose symbols will be read when 
needed. 


(gdb) info functions 
Prints the names and data types of all defined functions. 
(gdb) info functions regexp 


Prints the names and data types of all defined functions whose names contain a match for regular 
expression regexp. Thus, ‘info fun step’ finds all functions whose names include step; 


‘info fun “step' finds those whose names start with step. 
(gdb) info variables 


Prints the names and data types of all variables that are declared outside of functions (i.e., excluding 
local variables). 


(gdb) info variables regexp 


Prints the names and data types of all variables (except for local variables) whose names contain a 
match for regular expression regexp. 


Hey! Stop and look around 
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In GDB, we have three ways of interrupting the program flow and inspect what we need. We have 
Breakpoints, watchpoints and catch points. 


A breakpoint stops the execution at a particular location within the program. We have temporary 
breakpoints, regexp breakpoints and we could set conditional breakpoints. 


The usual breakpoint: 
(gdb) break <source>:<line> 


(gdb) break <source.c>:<function> 


(gdb) break 3 < This one stops at line 3 of the current source file being executed. 

(gdb) break <function> 

The temporary break point is a simple breakpoint that is deleted after it is hit. The command for this is: 
(gdb) tbreak <same format as breakpoint> 

The regexp breakpoint sets breakpoints at the functions matching the regexp provided 

(gdb) rbreak “city 

Conditional breakpoint, stops the execution of the program only if the condition is met. 

(gdb) b if strcmp(commands[0].synopsis,"*start") == 


Yes, you could use the C library functions as long as your program is linked against libc. You can enable 
or disable breakpoints with the following command: 


enable once -- Enable breakpoints for one hit 
enable delete -- Enable breakpoints and deletes when hit 
(gdb) enable once 1 (gdb) enable delete 1 


A watchpoint stops the execution when a particular memory location (or an expression involving one or 
more locations) changes value. Depending on your system, watchpoints may be implemented in 
software or hardware. GDB does software watchpointing by single-stepping your program and testing 
the variable's value each time, which is hundreds of times slower than normal execution. However, it’s 
really useful if you don’t have a clue of where the problem is in your program. 


The syntax for this command is: watch <expr> 
(gdb) watch commands[0] Watchpoint 1: commands [0] 


A catchpoint stops the execution when a particular event occurs. The event could be one of the 
following: 
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Raised signals may be caught: 

catch signal - all signals 

catch signal <signame> - a particular signal 
Raised exceptions may be caught: 

catch throw- all exceptions, when thrown 


catch throw <exceptname> - a particular exception, when thrown catch catch - all exceptions, when 
caught 


catch catch <exceptname> - a particular exception, when caught 
Thread or process events may be caught: 


catch thread_ start - any threads, just after creation catch thread_exit - any threads, just 
before expiration catch thread_join- any threads, just after joins 


Process events may be caught: 


catch start - any processes, just after creation catch exit - any processes, just before expiration 
catch fork - calls to fork() 


catch vfork - calls to vfork() 

catch exec - calls to exec() 

Dynamically-linked library events may be caught: 

catch load - loads of any library 

catch load <libname> _ - loads of a particular library catch unload - unloads of any library 
catch unload <libname> - unloads of a particular library 

The act of your program's execution stopping may also be caught: 

catch stop 

C++ exceptions may be caught: 

catch throw - all exceptions, when thrown catch catch - all exceptions, when caught 


You can enable and delete breakpoints, watchpoints and catchpoints with the enable and delete 
command. 


Proposed Exercises: 


Set a breakpoint that gets triggered when the size of a city is greater than 20 citizens. 
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Do you know what functions modify the game data type? Find out using a watchpoint. 
Display leader's and city names when a city’s data type is being read or modified by the program. 
When is a government during a revolution being set? Use a watchpoint or a conditional breakpoint. 


For any questions or issues, just use the forum 
http://bsdmag.org/forums/forum/application-debugging-and-troubleshooting/ 


Have fun. 
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Introduction to The JDB 
The Java Debugger 


“The Java Debugger (JDB) is a simple command-line debugger for Java classes. The jdb command and 
its options call the JDB. The jdb command demonstrates the Java Platform Debugger Architecture 
(JDBA) and provides inspection and debugging of a local or remote Java Virtual Machine (JVM). See 
Java Platform Debugger Architecture (JDBA) at 
http://docs.oracle.com/javase/8/docs/technotes/guides/jpda/index.html” 


JDB, like GDB, is a command line debug tool that will help us find bugs in our code (it is provided by 
openjdk), hopefully with little effort. The application that will be used to demonstrate the usefulness of 
JDB will follow the setting as the last one. The application we are going to use is called Freecol: 


"The FreeCol team aims to create an Open-Source version of Colonization (released under the 
GPL). At first, we'll try to make an exact clone of Colonization. " 








Like the last time we used GDB, we need the Freecol application sources to be compiled with extra 
debugging symbols (-g flag to the java compiler). 


Requirements: You must download the following package : 
http://prdownloads.sourceforge.net/freecol/freecol- 0.10.7-src.zip?download 





You must have the openjdk7 package installed 3- You must have apache-ant installed 
Initial Steps: Go to the folder where you have extracted the Freecol sources 


Type ‘ant’. This will start building the Freecol application with debugging information (you will see as was 
in GDB, the -g flag is used by the compiler if you look at the build.xml ). 


This will take some minutes and you will have anew FreeCol.jar file 
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/usr/home/cnei ra/workshopj db/ freecol/srce/net/sf/freecol/common/mode 
: warning: unmappable character for encoding ASCII 


* latitude. Thus, -30 equals 3677N, and 46 equals 4677S. 


rtd 
ra 
a 


- usr/home/cnei ra/workshopj db/freecol/src/net/st/freecol/common/mode 
Re Chant eee lel es nist ds encoding ASCII 
ac] ee aha |: et ee ee eed ee ce ee eee 
Foal 
ac] /usr/home/cneira/workshopjdb/freecol/sre/net/sf/freecol/common/mode 
ava:460: ng: unmappable character for encoding ASCII 
hele * latitude. Thus, -36 equals 3677N, and 46 equals 46775 
rhe al ; 
TP Loa 9 


warnings 


properties: 


Coa iG eee 


properties: 


[exec] Result: 
Test: 


he 


(jar) Building jar: /usr/home/cnei ra/workshopj db/freecol 


D SUCCESSFUL 
Ll time: 53 seconds 





There are two ways to start debugging using JDB. 


The first one is to give JDB the initial class (the one that has the main function) and start from there. As 
the JVM has not started, you must type ‘run’ to start the program. We will use the second approach that 
is to start a JVM and connect to it: 


$ java -jar -Xmx256M -Xdebug -Xrunjdwp:transport=dt_socket,server=y,address=6000 FreeCol.jar 
--no-intro 





Here, | use the --no-intro parameter because there is an issue with Freecol where the intro movie 
freezes the application. 


Then, we need to attach to this running JVM with the following command: 
$ jdb -attach localhost: 6000 

Now, let's start with some basic commands. 

Setting breakpoints : 


First, we need to know how to set breakpoints: 
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stop in <class-name>.<method-name> Sfop on entry to the given method. 
stop at <class-name>:<line-number> Sfop at the given line. 

clear <class-name>.<method-name> Remove the specified breakpoint. 
clear <class-name>:<line-number> Remove the specified breakpoint. 
Stepping in Stepping out 


When you hit a breakpoint, you either take a look at the data or continue executing the program. Here is 
the syntax for continuing, stepping, etc. 


cont 
As the name suggests, continue the execution after you hit the breakpoint. 
step 


Execute the current line . If the breakpoint is at a method call, it will stop the program at the method 
entry (same as GDB). 


next 


Execute the current line. If the breakpoint is at a method call, do not stop at the method entry (same as 
GDB). 


step up 
Execute until the current method returns to its caller (in GDB, the command is called finish). 
Take a look at the source code: 


Like in GDB, this command bears the same name. You need to execute jdb where the source is located 
or use the command in the JDB and provide the directory with the source code. 


list Lists 10 lines starting 4 before the current line. 

list <linenumber> Lists 10 lines starting 4 before the given line. 

list <method_name> Lists the first 10 lines of the given method. 

Taking a peek 

To look at the values of variables or expressions, we have the following commands: 
print <name> _— Prints the current value of the given variable. 

print <expression> Prints the value of the given expression. 


locals Prints the values of all variables local to the current method (In GDB, this one is called Info 
locals). 
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Calling for help 
The same as GDB, just type ‘help’. 
Option flags 


When you use the jdb command instead of the java command on the command line, the jdb command 
accepts similar options as those accepted by java command, including -D, -classpath, and -X options. 
The following list contains additional options that are accepted by the jdb command. 


-help 
Displays a help message. 
-sourcepath dir1:dir2: ... 


Uses the specified path to search for source files in the specified path. If this option is not specified, then 
use the default path of dot (.). 


-attach address 

Attaches the debugger to a running JVM with the default connection mechanism. 
-listen address 

Waits for a running JVM to connect to the specified address with a standard connector. 
-launch 


Starts the debugged application immediately upon startup of JDB. The -launch option removes the need 
for the run command. The debugged application is launched and then stopped just before the initial 
application class is loaded. At that point, you can set any necessary breakpoints and use the cont 
command to continue execution. 


-listconnectors 

Lists the connectors available in JVM. 

-connect connector-name:name1=value1 

Connects to the target JVM with the named connector and listed argument values. 
-dbgtrace [flags] 

Prints information for debugging the jdb command. 

-tclient 


Runs the application in the Java HotSpot VM client. 


31 


-tserver 
Runs the application in the Java HotSpot VM server. 
-Joption 


Passes option to the JVM, where, option is one of the options described on the reference page for the 
Java application launcher. For example, -J-Xms48m sets the startup memory to 48 MB. 


Exercise: 
Set a breakpoint when a new Colony is built, and print the stacktrace 
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Introduction to The JDB 
Start Debugging 


JDB, like GDB, is a cli-debug tool that will help us find bugs in our code, hopefully with little effort. 

The application that will be used to demonstrate the usefulness of JDB will follow the setting as the 
last one. The application we are going to use is called Freecol. 

"The FreeCol team aims to create an Open-Source version of Colonization (released under the GPL). At 
first, we'll try to make an exact clone of Colonization. " 

Like the last time we used GDB, we need the Freecol application sources to be compiled with extra 
debugging symbols (-g flag to the Java compiler). 


Requirements : 


1- You must download the following package : 
http://prdownloads.sourceforge.net/freecol/freecol-0.10.7-src.zip?download 
2- You must have the openjdk7 package installed. 

3- You must have apache-ant installed 


Initial Steps: 


1- Go to the folder where you have extracted the Freecol sources. 

2- Type ’ant’. This will start building the Freecol application with debugging information (you will see as 
was in GDB, the -g flag being used by the compiler if you look at the build.xml). 

3- This will take some minutes and you will have a new FreeCol.jar file. 


Start debugging 

There are two ways to start debugging using JDB. 

The first one is to give JDB the initial class (the one that has the main function) and start from there. As 
the JVM has not started, you must type ‘run’ to start the program. We will use the second approach 


that is to start a JVM and connect to it: 


S$ java -jar -Xmx256M -Xdebug 


=SEUN TOW? Pralisport=di socket, server=y,address=5000 PreéCol. jar =——no-intro 
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Here, | use the --no-intro parameter because there is an issue with Freecol where the intro movie 
freezes the application. 


Then, we need to connect it to the running JVM with the following command: 
$ jdb -attach localhost:6000 

Now, let's start with some basic commands. 

Setting breakpoints: 

First, we need to know how to set breakpoints: 


stop in <class-name>.<method-name> Stop on entry to the given method. 


stop at <class-name>:<line-number> Stop at the given line. 
clear <class-name>.<method-name> Remove the specified breakpoint. 
clear <class-name>:<line-number> Remove the specified breakpoint. 


Stepping in Stepping out 
When you hit a breakpoint, you either take a look at the data or continue executing the program. Here is 
the syntax for continuing, steping, etc.: 


cont 


As the name suggests, continue the execution after you hit the breakpoint. 
step 


Execute the current line . If the breakpoint is at a method call, it will stop the program at the method 
entry (same as GDB). 
next 


Execute the current line. If the breakpoint is at a method call, do not stop at the method entry (same as 
GDB). 
step up 


Execute until the current method returns to its caller (In GDB, the command is called finish). 
Take a look at the source code: 

Like in GDB, this command bears the same name. 

list Lists 10 lines starting 4 before the current line. 

list <linenumber> Lists 10 lines starting 4 before the given line. 


list <method_name> Lists the first 10 lines of the given method. 


Taking a peek 
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To look at the values of variables or expressions, we have the following commands: 


print <name> Prints the current value of the given variable. 

print <expression> Prints the value of the given expression. 

locals Prints the values of all variables local to the current method. 
Calling help 


The same as GDB, just type ‘help’. 
Here is a table of the command syntax between GDB and JDB. 


JDB commands GDB commands 
step Ss 
next n 
cont Cc 
stop in/at b 
clear info b 
step up finish 
up/down f 
where bt 
print/dump p 
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Working with Core 
Dumps in GDB 


What is a core dump? 


Acore dump or core file is produced when a signal (man signal) indicates that is not possible to continue 
executing the program because of an error (for example SIGSEV). A core file contains the current state 
of the program when it was executing (the stack of each thread, contents of the CPU registers, the 
values of global and static variables, etc..), and if we are lucky and the program that dumped core was 
compiled with debug options, the core file will have information about the source code and lines. A core 
file is used to do a postmortem analysis of what happened to our application. If we have the same 
versions of libraries between hosts, you could take a core produced, let’s say on a production machine, 
and analyze it in another machine that has the same libraries and versions (it is a must. Otherwise, your 
core file analysis will be invalid). 


Allowing to dump cores 


For the examples, we are going to use FreeBSD 10.0-STABLE to check if your installation is enabled to 
generate core files. You could execute the following commands as root: 


# sysctl -a |grep kern.coredump kern.coredump: 0 
# sysctl kern.coredump=1 kern.coredump: 0 -> 1 
# sysctl -a | grep kern.coredump kern.coredump: 1 


kern.coredump = 0 means that the core files will not be generated. Thus, use the sysctl (man sysctl ) 
facilities to change that, and make it permanent by editing /etc/sysctl.conf (man sysctl.conf). 


You could see the difference by just executing this simple program: 
#include<stdio.h> 

#include<stdlib.h> 

int Maan () 1 abori()s | 


Just name it test.c or whatever name you want. Then, do a make test or make <the name you have 
chosen> and you will see it core dumping and leaving a test.core file or none if you did not apply the 
changes to the kern.coredump variable. The abort() causes the program to abnormally terminate hence 
generating a core file. 
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Working with the core file 


Now that we have a core file, to take a look at it and start debugging, we just type: 


% gdb test test.core 


6 gdb <name of the executable must be in your path> <core file> 


3 e-ll eeoe 2 -: 
NU gdb 6.1.1 [FreeBSD] 
opyright 2004 Free Software Foundation, Inc. 
DB is free software, covered by the GNU General Public License, and you are 
elcome to change it and/or distribute copies of it under certain conditions. 
fe) #0 eS a ee 
here is absolutely no warranty for GDB. Type “show warranty” for details. 
his GDB was configured as “amd64-sarcel-freebsd”...(no debugging symbols found)... 


ore waS generated by ‘test’. 

rogram terminated with signal 6, Aborted. 

reading symbols from /lib/libc.so.7...(no debugging symbols found)...done. 
oaded symbols for /lib/libc.so.7 

heading symbols from /lib/Id-elf.so.1...(no debugging symbols found)...done. 
oaded symbols for /lib/ld-elf.so.1 

. a in kill © from /lib/libc.so.7 

ele 





As you see, there are no debugging symbols. | used the make command which took my CFLAGS from 
the 


/etc/make.conf file, and there, | did not have the —g flag set. 
Debugging without source code 


If debugging sometimes with the source code proves cumbersome, definitely, without the source code 
could be complicated. But let’s try debugging without it! 


We need to introduce the concept of Application Binary Interface. An ABI defines how system calls 
parameters are passed (which registers contain which argument), how functions are called and in which 
binary format information should be passed from one program component to another. FreeBSD ABI 
conforms to these specifications: 


http://Awww.x86-64.org/documentation/abi.pdf 


http:/Awww.sco.com/developers/devspecs/abi386-4.pdf 


Both documents are a good reading. You could also take a look at the developer’s handbook: 
http://www.nl.freebsd.org/doc/en/books/developers-handbook/x86-system-calls.html 


Now, maybe we are in a better position to debug some core files. Let’s try this simple program: 
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A a - 
#include<stdlib.h> 


int calling_convention_test(int a,int b,int c, int d) 


printf("printing a: %d b : Sd c: Md d: Md\n",a ,b,c,d); 
e0e 


int main() 


int res; 
res = calling_convention_test(1,2,3,4); 


abort(); 





I’m on x86_ 64 architecture. So, let’s see page 18, section 3.2.3 Parameter passing. In a nutshell, it says 
that integer parameters (CLASS TYPE INTEGERS) will be passed using registers 
%rdi, Yrsi, Yrdx, Y%rcx,%r8 and %r9. To check that, compile the program by typing: 


S cc <source.c> -o <exec name> 
Then, run the executable program that will get you a core file and load this into GDB as usual. 
$ gdb <exec> <core file> 


Now, we will use the disassemble command (disas is the short form). This command dumps a range of 
memory as machine instructions. The default memory range is the function surrounding the program 
counter of the selected frame. Two arguments are interpreted as a range of memory to dump. By default, 


GDB uses the AT&T notation (http://en.wikipedia.org/wiki/X86_assembly_ language). If we want to 
disassemble a specific function, just type: 


(gdb) disas <function name> 
We will always have a main function, but what about the names of the other ones? You could type: 
(gdb) info func 


And you will get all function names. This works if the executable has not been stripped (man 1 strip). 
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7 gdb checkparameters checkparameters.core 
NU gdb oe ee 
opyright 2004 Free Software Foundation, Inc. 
DB is free software, covered by the GNU General Public License, and you are 
elcome to change it and/or distribute copies of it under certain conditions. 
ype “show copying” to see the conditions. 
here is absolutely no warranty for GDB. Type “show warranty” for details. 
his GDB was configured as “and64-marcel-freebsd”...(no debugging symbols found)... 
ore was generated by ‘checkparameters' 
rogram terminated with signal 6, Aborted. 
Reading symbols from /lib/libc.so.7...(no debugging syabols found)...done. 
oaded sygsbols for /lib/libc.so.7 
eading symbols from /lib/ld-elf.so.1...(no debugging symbols found)...done. 
arti syabols for /lib/ld-elf.so.1 
0x00000008009Sd0ea in kill Q) from /lib/libc.so.7 
Vttlep mene 
) Ox000000080095Sd0ea in kill ( from /lib/libc.so.7 
0x000000080095b819 in abort () from /lib/libc.so 
0x00000000004007aa in sain () 
(gdb) f 2 
0x00000000004007aa in sain () 
(gdb) i lo 
No symbol table info available. 
gdb) i main 
Tmt ee ee Pt i 2 fc 
gdb) disas sain 
usp of asseabler code for function sain: 
)x0000000000400780 <main+0>: push tet 
1x0000000000400781 <main+l1>: Tey pie ett 
)x0000000000400784 <main+4>: ROV pet Same rk 
1x0000000000400789 <main+9>: mov pith ee oe 
)x0000000000400/78e <main+14>: BOV $0x2 ,%edx 
x0000000000400793 <main+19>: BOY $0x3 ,%ecx 
x0000000000400798 <main+24>: s0oVv pte me igi) 
7x000000000040079e <main+30>: Piel %eax ,%eax 
x00000000004007a0 <main+32>: URC C Ut lla a tilee he 
x00000000004007a5 <main+37>: callq 0x40050c <abort@p/t> 
x00000000004007aa <main+42>: nopw Ox0(*rax,¥rax,1) 
nd of asseabler dusp. 


eed i asseabler rts ‘> alee ps calling_ convention_test: 
00000000004007b0 <calling_convention_test+0>: push rbp 
ete: 00000004007b1 <calling_convention_test+ eee ft tet me ges 


Dx00000000004007b4 <calling_convention_test+4>: so *ecx , %r8d 
' ’ 7 <calling_convention_test¢7>: sov Xedx ,¥ecx 
“asl ing_convention_test¢9>: sov oT Ta hs 
» <cal ling_convention_test+¢1i>: teat tT DS 
<cal ling _convention_test¢13>: Tet pet Pim ore 
2? <cal ling_convention_test+18>: xor br bt 
) ere SRR Mle ee oe eal callq Ox4004dc <printfépit- 
a's Weel Me eiel ee eS BOV bit ele Tbs 
000000004007ce <calling_convention_test+30>: et b tare 
0x00000000004007cf <calling_convention_test+31>: fetes) 
nd of assesbler duap. 
) disas sain 0x04007cf 
usp of assesbler code from 0x400780 to Ox4007cf: 
0x0000000000400780 <sain+0>: thls leit 
Dx0000000000400781 <sainel>: BOV lett gts) 
eeeey+4l 84 <aaine4>: Tea bth wrap bendy 
j <aaineo:: Tea 50x11, ¥es 
» <painel4>: Pea pty ery 
2 hot e Te $0x3 , ¥ecx 
} <maine24>: Tay) $0x4 , ¥r8d 
<main+30>: Pits par ee BS 
' ‘ <main+32>: callq Ox4004dc <printfépit> 
0x00000000004007a5 <main+37>: RC MC tb lil@eslocee sl bes 
xD000000000400/aa <main+42>: nopw Duele) ee eh eee 
0x00000000004007b0 <calling_convention_test+0>: push &rbp 
0x00000000004007b1 <calling_convention_test+l>: mov tet ees 
x00000000004007b4 <calling_convention_test+4>: sov i.e gle 
Lee eeeeeeeeepeeded ve NAR al mee eee lh ae ae | 
7b9 <calling_convention_test+9>: sov ee eb a 
<calling_convention_test¢1i>: gov %edi ,¥esi 
<cal ling_convention_test¢13>: aqv pe Shee oe 
2 <cal ling convention _test+18>: xor Te er bs 
rie PRR me eee oe val callq Ox4004de <printfépit- 
9 <cal ling _convention_test+25>: Test ie tae bs 
Hee OO?ce <calling_convention_test+30>: ott state 
eee Me pt lea le 
(gdb) § 


Figure 2. disas command on calling_convention_test 





Set a breakpoint at the calling convention_test function, and run the program from the top. 


0 
gdb) disas 


0x0000000000400784 in Ca 


i mesh ihm ce et 8s 


uep of assembler code for function calling_convention_test: 


OU Ee ree 
UU eee ree E 
x0000000000400784 
x0000000000400/785 
x0000000000400790 
PUNE ree 
PU e Ure) 
pee ress) 
MOUUUnn nue els 
x000000000040079F 
x00000000004007 a2 
x00000000004007a5 
x00000000004007a9 
COUUUU nuit Uist 
x00000000004007 ae 
eee UU Uneaten) 
x00000000004007b8 
UU Enea el tres «) 
x00000000004007bd 
x0000000000400/7c1 
x0000000000400/7c2 
x00000000004007c3 


<Calling_convention_test+0>: 
<Calling_convention_test+l>: sov 
<calling_convention_test+4>: sub 
Pie SORT Meee ibm tee. ee | 
Pie SMR Meee ims eee te 8 
<cal ling_convention_test+19>: 
<cal ling_convention_test+22>: 
Pe SRM alee eS 
<cal ling_convention_test+28>: 
<calling_convention_test+31>: 
<cal ling_convention_test+34>: 
<cal ling_convention_test+37>: 
<calling_convention_test+41>: 
Pie SRR Me ele tee Cee 
<calling_convention_test+46>: 
<calling_convention_test+51>: 
<calling_convention_teste+56>: 
<calling_convention_test+59>: 
<calling_convention_test+61:>: 
<cal ling_convention_test+65>: 
<calling_convention_test+66>: 
<calling_convention_test+67>: 


ates 


ieee) mee pt eRe) le 


IL Me nn tt 


x400846 <.rodata>: 


gdb) x/s 0x400846 


x400846 <.rodata>: 


1852404336 


“printing a: db : &d c: 


Ut LT Mere Bt Lee 


0x0 
oe 
te) 
0x2 


1 el 


Ox7ffttttfesbo 
ee eeeear s+! 


UU 


eee eee lS rae Pat 
0x80081c520 
0x246 
0x7 fffffffea30 
Ox7fffffffea4s 
Ox7 fff FFF Feazs 


al 


1 

1 vee ee eeee.') +!) 

epee eeeen s+! 

t 

'94328540 


Ty 
rhs 


BE ELT Pe Pa 
582 

ETAL Sree 
140737488349768 
POET ELS Ter 
1 


ie tet) 

etme ge) t) 

$0x20,%rsp 
0x400846 , %rax 

ta Sedi ,-0x4(%¥rbp) 
ta %esi ,-O0x8(%rbp) 
ty %edx, -Oxc(%rbp) 
Ty %ecx, -0x10(%rbp) 
Tay a ote as) 
ta Le tae) ee ths 
te -Oxc(%rbp) , %ecx 
tony -0x10(%rbp) ,%r8d 
ta ie poe eek 

tat ie = 

UR tee iD G(s tilee shes 
Tey pith el Te la 

Tay %eax ,-0x14(¥%rbp) 
tay %ecx ,¥eax 

eTele $0x20,%rsp 

ae Bet 
fakes 


nopw %cs:0x0(%rax,%rax,1) 


ee 


516 





Figure 3. Using disas command 


As you can see, our parameters in the registers are being taken from the stack. To see this ina 
friendlier interface, use the layout command: 


(gdb) layout asm (gdb) layout regs 


That will split the screen showing the registers as they change at the top of the screen, and the asm 
instructions at the bottom. 
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Register group: genera 
rax 0x600968 6293864 y 
0x4 4 d ee F 


te Pr. eel 1 
theese eee | Ox7 fff tft fesdo Fe Mt thee eee ees») eee ee eel |! 


. ‘ageeer th ae a tk eee reel 
ewe te bt Fs 
tee eeees Tt! 
‘pel | 
Ox206 pee: 
1p | 
ip ie) 0 
apie) LF 


aie i 
tLe joe 
vention test 


_ d 
onvention_test . th dc <printfé@p!t> 
Lah i. oe 8 " 1 , eecx 
ico pee te tas) 





Figure 4. Seeing machine code and register info at the same time 


If you see the contents of the registers specified in the AMD64 ABI (“%rdi,%rsi, %Yrdx, Yrcx,%r8 and %r9), 
you will see the contents of the arguments that are being passed to the function <main+9>. If you only 
want to check the registers without using the GUI, just type: 


(gdb) info regs 
Or just print the one you need to check, for example %rcx register: 


(gdb) p $rcex 


What about if the executable has been stripped? 


This is tougher to debug as we don't have the symbol tables available. The symbol tables have been 
stripped from the binary, which means, we do not have the function names, all references to the names 
are gone and we only have all the operations using memory addresses. That means, we cannot even 
set a breakpoint using main. However, all we need to know the address of our main function. 


Let’s take a look at another example program: 
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> 


> 
void seemegodown(int, int); 
int sain() 
7 


if 
int a, b; 
seemegodown(a,b); 


i 


void seemegodown(int a, int b) 
< 
i 


int C= 
memset(&c, ' 





Figure 5. Code example 2 


Compile it as usual, and strip it: 

~ ¢C Seqtaulter.c:—o segqraulier 
S Sttaip seqfaulter 

Now, let’s find where the main is at. 


(gdb) info file 
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yabols froma “/usr/home/cneira/workshop/segfaulter”™. 

reeBSD sultithreaded core duap file: 
‘/usr/home/cneira/workshop/segfaulter.core’, file type elf64-x86-64-freebsd. 
OOM OU ROLE e tte nie Hee Lette ROO Oe mm cere 
0x000000080061b000 - 0x000000080063b000 is load2 
0x000000080081b000 - 0x000000080081d000 is load3 
0x0000000800b8e000 - 0x0000000800b9a000 is load4 
0x0000000800b9a000 - 0x0000000800bc3000 is load5 
0x0000000800c00000 - 0x0000000801400000 is load6 
OU ee eee mr pattlt Ouse eee ee mel ttm cme ley Fd 
0x000000080081d190 - 0x00000008008221b4 is .hash in /lib/libc.so.7 
0x00000008008221b8 - 0x00000008008279c8 is .gnu.hash in /lib/libc.so.7 
0x00000008008279c8 - 0x00000008008399T8 is .dynsym in /lib/libc.so.7 
UOC Ot eet pe Meet COU UO O U8) 00) 2 eld hte a eee ed 
0x0000000800840890 - 0x0000000800842094 is .gnu.version in /lib/libc.so.7 
0x0000000800842098 - 0x0000000800842160 is .gqnu.version_d in /lib/libc.so.7 
0x0000000800842160 - 0x000000080084e190 is .rela.dyn in /lib/libc.so.7 
0x000000080084e190 - 0x0000000800853308 is .rela.pit in /lib/libc.so.7 
0x0000000800853308 - 0x000000080085331b is .init in /lib/libc.so.7 
0x000000080085331c - 0x000000080085697c is .plt in /lib/libc.so.7 
0x0000000800856980 - 0x000000080095fb58 .text in /lib/libc.so.7 
0x000000080095fb58 - 0x000000080095 fb66 .fini in /lib/libc.so.7 
0x000000080095 Tb80 OP COU Ue Ue Hele ler -rodata in /lib/libc.so.7 
0x000000080096c8&d4 - 0x0000000800972c08 is .eh_frame_hdr in /lib/libc.so.7 
0x0000000800972c08 - O0x000000080098d84c is .eh_frame in /lib/libc.so.7 
0x0000000800b8e000 - Ox0000000800b8e004 is .tdata in /lib/libc.so.7 
0x0000000800b8e010 OP COU UU OHO) tbe) t Eble .tbss in /lib/libc.so.7 
0x0000000800b8e010 - 0x0000000800b8e030 is .ctors in /lib/libc.so.7 
0x0000000800b8e030 - O0x0000000800b8e048 is .dtors in /lib/libc.so.7 
0x0000000800b8e048 - 0x0000000800b8e050 -jcr in /lib/libc.so.7 
Pee U UG ete elt eb Le 0x0000000800b92f98 -data.rel.ro in /lib/libc.so.7 
0x0000000800b92f98 - 0x0000000800b93138 is .dynamic in /lib/libc.so.7 
0x0000000800b93138 - 0x0000000800b93a48 is .got in /lib/libc.so.7 
0x0000000800b93a48 - 0x0000000800b95588 is .got.pit in /lib/libc.so.7 
0x0000000800b95590 OP OU O OHO) 08 ok he R hele .data in /lib/libc.so.7 
0x0000000800b991b0 - 0x0000000800bc2f08 is .bss in /lib/libc.so.7 
0x0000000800600158 - 0x0000000800600214 is .hash in /lib/ld-elf.so.1 
0x0000000800600218 - 0x00000008006002f4 is .gnu.hash in /lib/Id-elf.so.1 
Up ae te 8 C00) 108) 0 Oe | tO OULU Otel eee) .dynsya in /lib/id-elf.so.1 
PCO UCU UO Ue teh hy Meemer eee eerie teal Bs .dynstr in /lib/ld-elf.so.1 
0x00000008006006d4 - 0x000000080060070c -gnu.version in /lib/Id-elf.so.1 
0x0000000800600710 - 0x00000008006007d8 is .gnu.version_d in /lib/ld-elf.so.1 
0x00000008006007d8 0x0000000800602080 is .rela.dyn in /lib/Id-elf.so.1 
OO UU UU teat tO OOOO ebb me ewe) ea A BTA er 2 beet 
0x0000000800615600 - 0x000000080061776a is .rodata in /lib/ld-elf.so.1 
0x000000080061776c - 0x0000000800618180 is .eh_frame_hdr in /lib/ld-elf.so.1 
0x0000000800618180 - 0x000000080061ac58 is .eh_frame in /lib/ld-elf.so.1 
0x000000080081b000 - 0x000000080081b008 is .dtors in /lib/ld-elf.so.1 
0x000000080081b010 - 0x000000080081b318 is .data.rel.ro in /lib/ld-elf.so.1 
0x000000080081b318 - 0x000000080081b458 is .dynamic in /lib/Id-elf.so.1 
0x000000080081b458 - 0x000000080081b550 is .got in /lib/ld-elf.so.1 
0x000000080081b550 - 0x000000080081b568 is .got.plt in /lib/ld-elf.so.1 
0x000000080081b570 - 0x000000080081b9e0 is .data in /lib/ld-elf.so.1 
0x000000080081b9e0 - 0x000000080081c770 is .bss in /lib/ld-elf.so.1 

local exec file: 
‘/usr/home/cneira/workshop/segfaulter’, file type elf64-x86-64-freebsd. 
Entry point: 0x4004e0 
0x0000000000400200 - 0x0000000000400211 is .interp 
0x0000000000400214 - 0x0000000000400244 is .note.tag 
0x0000000000400248 - 0x0000000000400280 is .hash 
OULU CUO HOU OPe TOME OD CeCe COO OOO tele Oe me mmr tte 
0x00000000004002b0 - 0x0000000000400388 is .dynsys 
0x0000000000400388 - 0x00000000004003e0 is .dynstr 
Pee Ue ee eer eee Te 0x00000000004003f2 1s .agnu.version 
0x00000000004003F8 - 0x0000000000400418 is .gnu.version_r 


Figure 6. Info file command 





There is the Entry point: 0x4004e0 
Let’s set a breakpoint at that memory area: (gdb) b *Ox4004e0 


Then, as we know that the program text area (code) is between 0x4004e0 and 0x4007a8. We could 
disassemble the range between those addresses (not really a thing you will want to do in a real 
application) other option that is nicer is just to display the program counter $pc the next couple of 
instructions every time you go down one instruction (nexti). 


(gdb) display /i Spc 


“ 


“ 


alah 
arb 
eke 


bo 


4 


r 
eee al 


wr emt tt are y 
ler code froa 0x4004e0 to 0x4007a8: 
4e0 <exit@p)t+20>: push &rbp 
Mel <exit@p]t+21>: foie 
4 <exit@p]t+24>: tek) 
6 <exit@p]t+26>: tee 
<exi t@p]t+28>: “r13 
Sebo he cle %r12 
eke hes pare 
a eke hee ee pe bs 
a eke hee tc ee! 
Po ehe hee (14) .Sr1$ 
4 <exitép!t+40 tebe ie 


s 
=] 


a=] 
— 


i a -—e 
“sea g fs 


a ne ee 


seocoects 
-* 


7 <exiv@p)t+43>: elle etme iol me tm ee! 
me ahs bes te $0x0,0x20057c(¥rip) Pte tir tiea ie nae ied 


eke he Sls ape ibih Mea + anes ber.b es 
6 <exit@p]t+58>: ee 00573(¥%rip) # 0x600a80 <envirom 
To ehe hele $0x8 ,.%r14 
<exi t@plt+ - ie hl le kle| 
<exit@p]t+72>: 0x400$3b <exit@pit+ill- 
exitép]t+74>: (%r14) ,%rax 
Soha he. 5 ree eb 
<¢xi t@p)t+ - its 0x400523 <exit@p)t+87> 
eke hes eee ape tbe Met ala hese bts 
4 PT ehe hes. oe ik 
3 <exit@p]t+87>: Rrax,0x200S36(¥rip) # 0x600a60 <__prognase> 
a <exit@p]t+94>: 0x400$30 <ex1t@p1t+100> 
<€x1 TOp1t+96o: teh 


iw waa 7 


= 


(Srax) ,%cl 
ead 
0x400520 <exit@pit+84> 


a 


<exi t@pit+96> 
<exit@p)t+lli-: nn i eb 
ie ahs hao ble < seh 
ie eke he ete : Ox400S4f <exit@plt+13b> 
<@x1 tOp]t+121L-: na eae eh 
<@x1t@p1t+124>: oe 0x40049c «<atexit@p]t> 
<ex1t@p]t+129>: pL) 0x400$54 <exit@p]t+136> 
<exi t@p]t+131L-: i Ox4004ac <_init_tls@p 
<exi tép]t+136>: 0 pit re kt 
is ee eb 
2 ehe bee Cee : 
> <exi tO@p]t+150>: 





Figure 7. disas command addresses range 
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por be bra p 


Petre ety 
<exi t@p11+623>: 
re ae Ren Pde 
<exit@p1t+631>: 
<€xi t@p11+636>: 
<exi t@p1t+640>: 
<exi t@p1t+641>: 
<exi t@p1t+642>: 
<exi t@p1t+644>: 
<exi t@p11t+645>: 
<ex1 t@p1t+648>: 
mabe shes bese 
<exi t@p1t+657>: 
a> ake hes ee 
<e€xi t@p1t+671>: 
<exi t@p1t+674>: 
<€xi t@p1t+677>: 
> aR Bens 
<exit@p1t+683>: 
<€xi t@p11+686>: 
<e€xi t@p1t+689>: 
<exi t@p1t+691>: 
<exi t@p1t+696>: 


is 
%rsp,%rbp 
pt elim tate 
-0x4(%rbp) ,%edi 
Sty et 6 158) ete 
0x400750 <exit@p1t+644> 
bith tee sr ba 
$0x10,%rsp 
ias)s 


= ae be 

%rbp 

%rsp,%rbp 

$0x10 ,%rsp 
pice. ee be 
$0x400 , %rdx 
-Oxc(%rbp) ,%rcx 
%edi , -0x4(%rbp) 
hee Te ase) 
Sty cee igs) ett 
Ste te ies) ee 
%esi,-Oxc(%rbp) 
ercx,%rdi 
poh ee he 
0x4004bc <memset@p!t> 
$0x10,%rsp 

tas)e) 





peal bothath i 
- 70 . 


Here is our function (notice the call to memset in there) <exit@plt+644>. 
Useful commands for looking at machine code: 


Apart for the ones described earlier, you can use these ones to take a look at what is happening in your 
program. 


Examining memory 


Use the x command to examine memory. The syntax for the x command is x/FMT ADDRESS. The FMT 
field is a count followed by a format letter and a size letter. Use the help command ‘help x' to see them 
all. The ADDRESS argument can either be a symbol name, such as a variable, or a memory address. 


Examine the variable as a string: 

(gdb) x/s <var> 

Examine the variable as a character: 
(gdb) x/c <var> 

Examine the variable as 4 characters: 
(gdb) x/4c <var> 

Examine the first 32 bits of the variable: 
(gdb) x/t <var> 


Examine the first 24 bytes of the variable in hex: 
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(qd): «/Sx-<var> 
Print the instructions at the memory area specified (gdb) x/i  <addr> 
Stepping 


As we inspect the machine code, we don’t have the source code. The commands that we usually use: 
next (n) and step(s) have their equivalents named with the ‘I’ suffix (for instruction): 


nexti (ni) 


stepi (si) 


Question: 


Based on the documentation, describe the GDB session steps that you will use to check the parameters 
being passed to the memset function, and the user-defined function of the program on example program 
2: 
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Introduction to 
Dtrace 


DTrace, or dynamic tracing, was first available in the Solaris 10 3/05 around 2005. DTrace is now 
available in FreeBSD beginning from 7.1 and Mac OS X from 10.5 (Leopard). DTrace differs from 
traditional tools in that code is instrumented dynamically (that means you can peek at the program 
without recompiling). 


From the handbook: https://www.freebsd.org/doc/en_US.1SO8859- 1/books/handbook/dtrace.html 


“The FreeBSD implementation provides full support for kernel DTrace and experimental support for 
userland DTrace. Userland DTrace allows users to perform function boundary tracing for userland 
programs using the pid provider, and to insert static probes into userland programs for later tracing. 
Some ports, such as databases/postgres-server and lang/php5 have a DTrace option to enable static 
probes. FreeBSD 10.0-RELEASE has reasonably good userland DTrace support, but it is not considered 
production ready. In particular, it is possible to crash traced programs.” 


Requirements 


I’m running FreeBSD 10.0-STABLE where Dtrace is already available as a kernel module. Typing the 
following as root will let you know that you are ready to fire up some probes using dtrace: 


(t:~ % sudo dtrace -1 | we -1l 


neira@Next:~ % 





If this fails, you need to recompile your kernel and follow the instructions from the handbook in here: 
https://www.freebsd.org/doc/en_US.ISQO8859-1/books/handbook/dtrace-enable.html 


Why do | care about DTrace? 


If you want to understand what is happening in your software without needing recompiled special 
versions of your applications (lots of debug messages, maybe recompile with debug flags to use a 
debugger?) and also centralize all your current instrumentation tools into just one, then you should care 
about DTrace. 


Some Features of DTrace. 


DTrace is dynamic: probes are enabled only when you need them. 
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No code is present for inactive probes. 

There is no performance degradation when you are not using DTrace. 

When the dtrace command exits, all probes are disabled and instrumentation removed. 
The system is reverted to its original state. 

DTrace is nondestructive. The system is not paused or quiesced. 

DTrace is designed to be efficient. No extra data are ever traced. 


Because of its safety and efficiency, DTrace can be used in production to solve real problems in real 
time. 


Predicates: A logical predicate mechanism allows actions to be taken only when user- specified 
conditions are met. Unwanted data is discarded at the source—never retained, copied, or stored. 


A high-level control language: DTrace is equipped with an expressive C-like scripting language known as 
D . It supports all ANSI C operators, which may be familiar to you and reduce your learning curve, and 
allows access to the kernel’s variables and native types. D offers user-defined variables, including global 
variables, thread-local variables, and associative arrays, and it supports pointer dereferencing. This, 
coupled with the runtime safety mechanisms of DTrace, makes it admirable 


Trying DTrace : 
We will try a default script that comes with our FreeBSD installation. Go to 


/usr/share/dtrace/toolkit and execute the script called procsystime. This script “only processes system 
Call time details.” Notice the only. 


ooté@bsd: /usr/share/dtrace/toolkit # ./procsystime 
racing... Hit Ctrl-C to end... 
8 


lapsed Times for all processes, 


> ee TIME (ns) 
Sigreturn ew A) 
sigaction 4-6 Y | 
ee: 15745 
getpid pair BI 
__sysctl 39645 
munmap 47318 
getsockopt oe Eee / 
mmap 58251 

read Se Th) 
sigprocmask Pane Ys 
ioctl Ser 

clock _gettime 683811 
write Bree) 
_umtx_op Bi PRB Belt 
select 3014173473 





ootébsd: /usr/share/dtrace/toolkit # J 
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This one is pretty handy right away. Imagine what you could do with some imagination. Let’s see this 
one liner: which processes are executing the most system calls? 


oot@bsd:/usr/share/dtrace/toolkit # dtrace -n ‘syscall:::entry { @[pid, execname] = count();}' 
itrace: description ‘syscall:::entry ' matched 536 probes 


1398 preload 
1377 sendmail 
2118 sshd 
2273 dtrace 
TOD eee tide ee tsa 





Pretty impressive, isn't it? At this moment, you are wondering how all these years you have lived without 
DTrace. Using truss, strace ,lsof even gdb now seems pretty lame. Well, gdb is not so lame now. 


DTrace Scripting: 


DTrace scripts are written in the D language. You could take a look at this reference 
http://dlang.org/spec.html. 


Now, let’s write our first probe. A DTrace script has the following structure: 
Your probes 

/ predicate (usually you will create a filter here) / 

{ 

What are you going to do when you hit a probe 

} 


Let’s create a simple one to get used to the syntax, and later, we will dissect it line by line, ok? This one 
does not have a predicate, so it will capture all that the probe is asking for. 


A predicate is a conditional statement (IF statement if you like). 





syscall:::ent a> your 
ry probe 
(@[pid,execname] = >> your 


} count(): action 
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Save this to a file called example1.d . Then, execute the script by typing: dtrace —s example1.d 
The probe section has the following syntax: 


provider :module: function: name 
What every section means: 


Provider The name of the DTrace provider that is publishing this 
probe. The provider name typically corresponds to the 
name of the DTrace kernel module that performs the 
instrumentation to enable the probe. 


Module If this probe corresponds to a specific program location, 
the name of the module in which the probe is located. 
This name is either the name of a kernel module or the 
name of a user library. 


Function a 
If this probe corresponds to a specific program location, 
the name of the program function in which the probe is 
located. 

Name The final component of the probe name is a name that 


gives you some idea of the probe’s semantic meaning, 
such as BEGIN or ENDS. In this case, the probe says that 
is the entry of a function call. 


What providers are available to us in FreeBSD? Well, you should dig in and see what you need. 


root@bsd:"/dtracescripts # dtrace -1 i head -1 
eh EUR BID: MODULE FUNCTION NAME 
root@bsd:”~/dtracescripts # dtrace -1 i head -Z2 
8h PROVIDER EULER FUNCTION NAME 
i eS tet BEGIN 
root@bsd:”~/dtracescripts dtrace -l1 i: head -5 
Bh wee Bi, MODULE FUNCTION NAME 
i eS Dele BEGIN 
2 dtrace END 
3 dtrace Pdi 


mn kernel camstatusentrycomp entry 


root@bsd:~/dtracescripts dtrace -l i 
EER BID: MODULE FUNCTION NAME 


dtrace BEGIN 
dtrace END 
dtrace PP 
fbt kernel camstatusentrycomp entry 
eine kernel ee tee Ad Bete 
ein kernel cam_compat_handle_Ox17 entry 
root@bsd:~/dtracescripts 





Now to this line: 


@[pid,execname] = count(); 
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This is called an aggregation, and is denoted by the @ special character. Aggregations are global in your 
Dtrace scripts. The syntax of an aggregation is as indicated below: 


@name[ keys 


aggfunc ( args 


Name: The name you choose for the aggregation. 


Keys: Comma-separated list of D expressions (in this case, we are asking for pid and the name of 
executable which triggers the probe). 


Aggfunc: This is one of the DIrace aggregating functions. Moreover, args is a comma-separated list of 
arguments appropriate for the aggregating function. 


Here are the aggregation functions available: 


Function 
Name 


count 
sum 
avg 
min 


max 


Iquantize 


quantize 


Arguments 


none 
scalar expression 
scalar expression 
scalar expression 


scalar expression 


scalar 
expression, 
lower bound, 
upper bound, 
step value 


scalar expression 


Result 


The number of times called. 

The total value of the specified expressions. 

The arithmetic average of the specified expressions. 
The smallest value among the specified expressions. 
The largest value among the specified expressions. 


A linear frequency distribution, sized by the specified 
range, of the values of the specified expressions. 
Increments the value in the highest bucket that is less 
than the specified expression. 


A power-of-two frequency distribution of the values of 
the specified expressions. Increments the value in the 
highest power-of-two bucket that is less than the 
specified expression. 
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Now, let’s add a predicate to the same script. If you looked at the output of the script you would have 
seen that it counted the system calls done by dtrace itself. Let’s filter that. 


oot@bsd:"/dtracescripts # dtrace -s examplei.d 
dtrace: script ‘examplel.d’ matched 536 probes 
Le 


1466 sh 
1244 syslogd 
1464 cron 
1467 preload 
eri ea ee 
1397 preload 
1466 dd 
aed 
1465 cron 
1467? vmstat 
1465 sh 
1363 cron 
‘oot@bsd:”/vdtracescripts # cat examplel.d 
syscall: :tentry 
execname t= “dtrace” / 
{ 


@lpid,execname] = count(): 





But how did | know that execname contained the name of the program being executed? Well, it is a 
built-in variable in DTrace. Here is a list of some Dtrace Built-in variables: 


DTrace Built-in Variables 
Type and Name Description 


The first ten input arguments to a probe represented as raw 64-bit integers. If 
int64_t argO, fewer than ten arguments are passed to the current probe, the remaining 
.., arg9 variables return zero. 


The typed arguments to the current probe, if any. The args|] array is accessed 
using an integer index, but each element is defined to be the type 
corresponding to the given probe argument. For example, if args[] is referenced 


args|] by a read(2) system call probe, args[0] is of type int, args[1] is of type void *, 
and args[2] is of type size_t. 

uintptr_t The program counter location of the current thread just before entering the 

caller current probe. 


The name of the current working directory of the process associated with the 
string cwd current thread. 


The enabled probe ID (EPID) for the current probe. This integer uniquely 


identifiers a particular probe that is enabled with a specific predicate and set of 


uint_t epid actions. 


int errno The error value returned by the last system call executed by this thread. 


52 


computations. 


uid_t uid The real user ID of the current process. 

uint64 t The current thread's saved user-mode register values at probe firing time. Use 
uregs|] of the uregs|]. 

uint64 t The current thread's active virtual machine register values at probe firing time. 
vmregs|] Use of the vmregs]]. 


The current value of a nanosecond timestamp counter that is virtualized to the 
amount of time that the current thread has been running on a CPU, minus the 


uint64_t time spent in DTrace predicates and actions. This counter increments from an 

vtimestamp arbitrary point in the past and should only be used for relative time 
computations. 

uint64 t The current number of nanoseconds since 00:00 Universal Coordinated Time, 

walltimestamp January 1, 1970. 


You could take a look at the full listing in this URL: 
http://docs.oracle.com/cd/E18752_01/html/819-5488/gcfpz.html 


Assignment: 


Pick any program that you are certain that uses the strcmp(3) C library function, and print the arguments 
passed to this function. Tip: first create a script, similar to the example, to check for the strcmp call 
being used in any the programs executing in your machine. 
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